Scaling Relationships in Back-Propagation Learning: Dependence on Training Set Size

We stud y th e amount of ti me needed to learn a fixed t raining se t in the "back-pro pagation" proced ure for learning in multi-layer neural network models. The task chosen was 32-bit parity, a highorder funct ion for wh ich memorization of specific inpu t-output pairs is necessary. For small t raining sets , the learning time is consistent with a ~-power law depen dence on the number of patterns in the t ra ining set. For larger training set s, t he learn ing t ime dive rges at a critical tra ining set size which appears to be related to the storage capacity of the network. There is now widespread interest in devising adaptive learning proce dures for massively-p arallel networks of neuron-like comput ing elements [1,2,10 ,11,12,17]. A prime example is the "back-p ropagat ion" learn ing procedure [8,9,6] for mu lti-l ayer feed-forward network s. In this procedure, a set of patterns is presented to the input layer of the network, and the networ k's output is computed by feed-forward pr opagation of the input signal. An error function is then obtained by calculating the mean squar ed difference between the network 's output and the des ired output for each pattern . T he connection strengths, or weights, of the network are then modified so as to minimize the erro r fun cti on according to a grad ient-descent rul e. This algorithm has displayed impress ive perfor man ce for small-scale prob lems, and it is now of great interest to determine how it will scale to larger , more d ifficult prob lems. T he question of sca ling can be approached from several different dir ect ions. Typically one wou ld ask how some measure of the network 's performance (such as the t raining t ime , or the fraction of pat terns classified correctly) scales wit h a paramete r describing the network , the task, or t he learning algorithm. Basic parameters describing the network include: the numb er of input units n, the number of hidden units h, and the number of layers 1. Related paramet ers include: the total number of possibl e inpu t patterns (2" for binary inp uts), and the storage capacity of the netwo rk, © 1987 Complex Systems Publications, Inc.